Skip to content

Fix three flaky CI tests: demo dev-signer start, sensor event time, netserver log - #766

Merged
LoopGlitch26 merged 2 commits into
mainfrom
agent/flaky-py39
Nov 13, 2026
Merged

LoopGlitch26 merged 2 commits into
mainfrom
agent/flaky-py39

Conversation

@LoopGlitch26

Copy link
Copy Markdown
Member

Three tests failed CI at random on unrelated pull requests. Each is a test bug, now fixed at its cause, with a regression test that fails on main.

How often, and where

Read from the logs of the 165 failed test jobs among the last 300 CI runs (2026-11-06 07:10 to 2026-11-12 14:09 UTC):

Test Failed jobs Where On main
test_demos.py::Demos::test_each_demo_verifies_and_runs_as_recorded 4 of 184 py3.9 Linux jobs, all 4 among the 21 since 11-12 03:44 py3.9 ubuntu-latest only no
test_sensor.py, the same 4 tests each time (TestImport x3, TestFalcoImport) 6 of 358 Windows jobs windows-latest, py3.9 (2) and py3.13 (4) once: 3e6eb28, run 43164829924
test_netserver.py::SilentClient::test_a_cut_connection_logs_no_traceback_but_a_handler_error_does 1 windows-latest py3.13, run 43169237820 no

1. The demo check called a dev signer that was still starting hung

All four failures are tracekit.sdk.autospawn.Hung: a dev signer (pid None) holds .../signer.lock but does not answer. Three came from guardrails and one from completeness, the demos that run on the auto-started dev signer. pid None means the signer had taken its lock but had not yet published endpoint.json: it was still starting when the client's 10 s (autospawn.ensure) ran out.

It was slow, not stuck. The failing --check runs took 18.1 to 19.0 s in all; passing ones on py3.9 Linux took 11 to 19 s. A signer that never served would have added up to 10 s more, because tracekit down waits that long for its pid.

Why it was slow: --check runs every demo at once, in a thread pool of cpu+4 = 8 on a 4-vCPU runner. Each demo starts several Python processes (the demo, its signer, CLI calls), so about nine signers start together, and py3.9 is the slowest interpreter. The failures began 10 hours after #605 grew dev.yaml from 31 to 163 lines and the policy engine. That is timing only: a signer's start time wasn't measured here.

Fix, in scripts/record_demos.py: for the dev signer's demos, the script starts the signer itself, as autospawn would. It then waits until the signer publishes endpoint.json (written once it serves) or exits. story.Signer already waits the same way for a configured signer's serving on line. The demo then finds the signer running. The 10 s in autospawn stays: a hook that gets no answer must still fail closed within its harness's timeout. tests/test_signer_dev.py still covers autospawn from cold, end to end.

Regression test: a sitecustomize makes the dev signer's start 11 s slower once it holds signer.lock. On main the completeness demo then fails with CI's Hung (pid None); here it passes.

2. The sensor tests' event time ran out an hour after import

TIME, the time of the tests' sensor events, was "an hour from now" at import, so at collection. The signer gives an event to the run registered by its time. A test that runs more than an hour after collection registers its run after TIME, and its events go to no run: [] != [('exec', 'call_7', False), ...].

The failing Windows jobs took 69 to 72 minutes, and all 6 reached test_sensor 59.8 to 61.1 minutes after collection. Of 135 sampled Windows jobs, the 129 that passed reached it at most 58.7 minutes after.

Fix: TIME is a fixed timestamp in 2099. Regression test: a run registered two hours from now still takes the event (the signer's clock is patched, as tests/factories.py does). On main it fails with 0 != 1.

3. The netserver test read the log before the server had written it

The connection's 0.3 s deadline keeps running while the server thread writes a handler's traceback. A client can therefore see the connection cut before the traceback is written, and the test then slept a fixed 0.3 s. CI captured only the traceback's first line.

Fix: the test waits until the server is done with each connection. Its shutdown_request runs after handle_error. With the traceback written 1 s late, main's test fails with CI's message, and this one passes.

Stability

On macOS, with another agent's full test run and my own loops loading the machine:

  • test_sensor.py and test_netserver.py, 20 runs on Python 3.11: 20 of 20 passed (51 tests each).
  • test_demos.py, 20 runs on 3.11: 20 of 20 passed. Every demo runs there, including all three on the dev signer.
  • The three files, 20 runs on Python 3.9.25: 20 of 20 passed (57 tests each). Note: on macOS, py3.9's time.monotonic() starts near 0 in each process, and an import sends no heartbeat in its process's first 30 s. So test_sensor.py run alone on macOS py3.9 fails on main too; here it runs after the other two files. CI's long full run is never under 30 s by then.
  • Full gate from the branch's worktree, make test-ts lint build test: passed. test-ts: 162 tests, 0 failed. Lint and build clean. pytest: 4,973 passed, 394 skipped, 10 xfailed, in 29:52.

Not checked

The Windows causes are argued from the code and CI logs; nothing ran on Windows. The demo script's wait relies on endpoint.json, which Windows' TCP dev transport writes atomically after it binds, as the Unix socket path does after binding.

…etserver log

- record_demos --check starts every demo at once; on py3.9 Linux a dev
  signer's first start then outlasted the 10 s a client waits for one it
  starts (autospawn.Hung, pid None). The script now starts the dev signer
  of its demos and waits until it publishes endpoint.json.
- test_sensor's event time was an hour after import; Windows CI reached
  the file more than an hour after collection, so the runs registered
  after their events. It is now a fixed 2099 time.
- test_netserver slept 0.3 s for the server's traceback, which the
  connection's deadline can outrun; it now waits until the server is done
  with each connection.

Each fix has a regression test that fails on main.

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
…running

Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
@LoopGlitch26
LoopGlitch26 merged commit 47559f9 into main Nov 13, 2026
34 checks passed
@LoopGlitch26
LoopGlitch26 deleted the agent/flaky-py39 branch November 13, 2026 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant